The demand for bandwidth in networking applications and faster network speeds are increasing faster than the memory bandwidth and processing power of the computer nodes that process the data traffic. The migration to 10 Gigabit Ethernet is also exacerbating the problem. Typically, communication via TCP/IP requires data copying operations that utilize valuable CPU and memory resources and add latency.
Remote Direct Memory Access (RDMA) technology over TCP addresses various bandwidth and processing power issues. With RDMA over TCP, a significant amount of protocol processing overhead may be moved to the Ethernet adapter, for example an RDMA Network Interface card (RNIC). Also, every incoming network packet contains enough information such that the packet may be placed directly into a final destination memory address. The property of direct data placement (DDP) possessed by RDMA eliminates the need for intermediate memory copies and other related memory and processor resource demands. It may also remove the need for data buffering on the NIC. In addition, the reduction of data copy operations that RDMA provides results in reduced latency.
With RDMA, a first computer may directly write information into a second computer's memory with very few demands on the bandwidth of the memory bus and the processing overhead of an associated CPU. Memory protection semantics may also be preserved. Therefore, RDMA, as well as Direct Memory Access in general, provides many benefits.
When reading from or writing to a memory location associated with a Memory Region (MR) or a Memory Window (MW), some applications may need to ensure that memory is accessible exactly one time or some limited number of times. For example, there are security concerns associated with leaving host memory exposed for extended periods of time. When managing host memory, excessive overhead costs may result from tearing down the registration of the MR or MW with the RNIC, assigning the resource to another consumer (e.g., another thread or another memory location), and registering the resource again with the RNIC. Tearing down the resource registration consumes software resources and makes resource recycling less efficient due to longer time required from the time the remote last uses the resources until they are removed and can become available for another use. Alternatively, if memory is registered with the RNIC while assigning the same memory to another consumer, undesired side effects of exposing resources while in transition may occur.
RDMA over TCP provides the DDP capability as described above. RDMA flexibly allows placement of information in the designated memory location even when the TCP segment carrying that information arrives at the destination out-of-order, resulting in a TCP hole. Support for a ‘one-shot’ resource usage when DDP service is requested for out-of-order frames complicates the control mechanism of the ‘one-shot’ or alternatively requires buffering till the TCP hole is plugged. Both of these options add cost and complexity and should be avoided if at all possible. Accordingly, there is a strong need in the DDP/RDMA art for less complicated memory control technology that requires little or no additional buffering.
Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with the present invention as set forth in the remainder of the present application with reference to the drawings.